Knowledge-Based Automatic Topic Identification

As the first step in an automated text summarization algorithm, this work presents a new method for automatically identifying the central ideas in a text based on a knowledge-based concept counting paradigm. To represent and generalize concepts, we use the hierarchical concept taxonomy WordNet. By setting appropriate cutoff values for such parameters as concept generality and child-to-parent frequency ratio, we control the amount and level of generality of concepts extracted from the text. 1 1 I n t r o d u c t i o n As the amount of text available online keeps growing, it becomes increasingly difficult for people to keep track of and locate the information of interest to them. To remedy the problem of information overload, a robust and automated text summarizer or information extrator is needed. Topic identification is one of two very impor tant steps in the process of summarizing a text; the second step is summary text generation. A topic is a particular subject that we write about or discuss. (Sinclair et al., 1987). To identify the topics of texts, Information Retrieval (IR) researchers use word frequency, cue word, location, and title-keyword techniques (Paice, 1990). Among these techniques, only word frequency counting can be used robustly across different domains; the other techniques rely on stereotypical text structure or the functional structures of specific domains. Underlying the use of word frequency is the assumption that the more a word is used in a text, the more impor tant it is in that text. This method 1This research was funded in part by ARPA under order number 8073, issued as Maryland Procurement Contract # MDA904-91-C-5224 and in part by the National Science Foundation Grant No. MIP 8902426. recognizes only the literal word forms and nothing else. Some morphological processing may help, but pronominalization and other forms of coreferentiality defeat simple word counting. Furthermore, straightforward word counting can be misleading since it misses conceptual generalizations. For example: "John bought some vegetables, fruit, bread, and milk." What would be the topic of this sentence? We can draw no conclusion by using word counting method; where the topic actually should be: "John bought some groceries." The problem is that word counting method misses the impor tant concepts behind those words: vegetables, fruit, etc. relates to groceries at the deeper level of semantics. In recognizing the inherent problem of the word counting method, recently people have started to use artificial intelligence techniques (Jacobs and ttau, 1990; Mauldin, 1991) and statistical techniques (Salton et al., 1994; Grefenstette, 1994) to incorporate the sementic relations among words into their applications. Following this trend, we have developed a new way to identify topics by counting concepts instead of words. 2 T h e P o w e r o f G e n e r a l i z a t i o n In order to count concept frequency, we employ a concept generalization taxonomy. Figure 1 shows a possible hierarchy for the concept digital computer. According to this hierarchy, if we find iaptop and hand-held computer, in a text, we can infer that the text is about portable computers, which is their parent concept. And if in addition, the text also mentions workstation and mainframe, it is reasonable to say that the topic of the text is related to digital computer. Using a hierarchy, the question is now how to find the most appropriate generalization. Clearly we cannot just use the leaf concepts since at this level we have gained no power from generalization. On the other hand, neither can we use the very top concept everything is a thing. We need a method of identifying the most appropriate concepts somewhere in middle of the taxonomy. Our current solution uses